{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "9e3a8796-edc8-43f2-94ad-fe4fb20d70ed",
   "metadata": {},
   "source": [
    "# Oracle Cloud Infrastructure Generative AI\n",
    "\n",
    "Oracle Cloud Infrastructure (OCI) Generative AI is a fully managed service that provides a set of state-of-the-art, customizable large language models (LLMs) that cover a wide range of use cases, and which is available through a single API.\n",
    "Using the OCI Generative AI service you can access ready-to-use pretrained models, or create and host your own fine-tuned custom models based on your own data on dedicated AI clusters. Detailed documentation of the service and API is available __[here](https://docs.oracle.com/en-us/iaas/Content/generative-ai/home.htm)__ and __[here](https://docs.oracle.com/en-us/iaas/api/#/en/generative-ai/20231130/)__.\n",
    "\n",
    "This notebook explains how to use OCI's Genrative AI models with LlamaIndex."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3802e8c4",
   "metadata": {},
   "source": [
    "## Setup\n",
    "\n",
    "If you're opening this Notebook on colab, you will probably need to install LlamaIndex 🦙."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bb0dd8c9",
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install llama-index-llms-oci-genai"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "544d49f9",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install llama-index"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c2921307",
   "metadata": {},
   "source": [
    "You will also need to install the OCI sdk"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "378d5179",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install -U oci"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "03d4024a",
   "metadata": {},
   "source": [
    "## Basic Usage\n",
    "\n",
    "Using LLMs offered by OCI Generative AI with LlamaIndex only requires you to initialize the OCIGenAI interface with your OCI endpoint, model ID, OCID, and authentication method."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8ead155e-b8bd-46f9-ab9b-28fc009361dd",
   "metadata": {},
   "source": [
    "#### Call `complete` with a prompt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "60be18ae-c957-4ac2-a58a-0652e18ee6d6",
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.llms.oci_genai import OCIGenAI\n",
    "\n",
    "llm = OCIGenAI(\n",
    "    model=\"MY_MODEL\",\n",
    "    service_endpoint=\"https://inference.generativeai.us-chicago-1.oci.oraclecloud.com\",\n",
    "    compartment_id=\"MY_OCID\",\n",
    ")\n",
    "\n",
    "resp = llm.complete(\"Paul Graham is \")\n",
    "print(resp)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "14831268-f90f-499d-9d86-925dbc88292b",
   "metadata": {},
   "source": [
    "#### Call `chat` with a list of messages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bbe29574-4af1-48d5-9739-f60652b6ce6c",
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.llms.oci_genai import OCIGenAI\n",
    "from llama_index.core.llms import ChatMessage\n",
    "\n",
    "messages = [\n",
    "    ChatMessage(\n",
    "        role=\"system\", content=\"You are a pirate with a colorful personality\"\n",
    "    ),\n",
    "    ChatMessage(role=\"user\", content=\"Tell me a story\"),\n",
    "]\n",
    "\n",
    "llm = OCIGenAI(\n",
    "    model=\"MY_MODEL\",\n",
    "    service_endpoint=\"https://inference.generativeai.us-chicago-1.oci.oraclecloud.com\",\n",
    "    compartment_id=\"MY_OCID\",\n",
    ")\n",
    "\n",
    "resp = llm.chat(messages)\n",
    "print(resp)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "52e1b3d0",
   "metadata": {},
   "source": [
    "#### Call `chat` with messages and images"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "90a5ae1f",
   "metadata": {},
   "outputs": [],
   "source": [
    "test_image_url = \"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d300fa4c",
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.llms.oci_genai import OCIGenAI\n",
    "from llama_index.core.llms import ChatMessage, TextBlock, ImageBlock\n",
    "\n",
    "messages = [\n",
    "    ChatMessage(\n",
    "        role=\"user\",\n",
    "        blocks=[\n",
    "            ImageBlock(url=test_image_url),\n",
    "            TextBlock(text=\"Describe the image in a few sentences.\"),\n",
    "        ],\n",
    "    )\n",
    "]\n",
    "\n",
    "llm = OCIGenAI(\n",
    "    model=\"meta.llama-3.2-90b-vision-instruct\",\n",
    "    service_endpoint=\"https://inference.generativeai.us-chicago-1.oci.oraclecloud.com\",\n",
    "    compartment_id=\"MY_OCID\",\n",
    ")\n",
    "\n",
    "resp = llm.chat(messages)\n",
    "print(resp.message.content)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ed5e894-4597-4911-a623-591560f72b82",
   "metadata": {},
   "source": [
    "## Streaming"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4cb7986f-aaed-42e2-abdd-f274f6d4fc59",
   "metadata": {},
   "source": [
    "Using `stream_complete` endpoint "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d43f17a2-0aeb-464b-a7a7-732ba5e8ef24",
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.llms.oci_genai import OCIGenAI\n",
    "\n",
    "llm = OCIGenAI(\n",
    "    model=\"MY_MODEL\",\n",
    "    service_endpoint=\"https://inference.generativeai.us-chicago-1.oci.oraclecloud.com\",\n",
    "    compartment_id=\"MY_OCID\",\n",
    ")\n",
    "\n",
    "resp = llm.stream_complete(\"Paul Graham is \")\n",
    "for r in resp:\n",
    "    print(r.delta, end=\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "40350dd8-3f50-4a2f-8545-5723942039bb",
   "metadata": {},
   "source": [
    "Using `stream_chat` endpoint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bc636e65-a67b-4dcd-ac60-b25abc9d8dbd",
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.llms.oci_genai import OCIGenAI\n",
    "from llama_index.core.llms import ChatMessage\n",
    "\n",
    "messages = [\n",
    "    ChatMessage(\n",
    "        role=\"system\", content=\"You are a pirate with a colorful personality\"\n",
    "    ),\n",
    "    ChatMessage(role=\"user\", content=\"Tell me a story\"),\n",
    "]\n",
    "\n",
    "llm = OCIGenAI(\n",
    "    model=\"MY_MODEL\",\n",
    "    service_endpoint=\"https://inference.generativeai.us-chicago-1.oci.oraclecloud.com\",\n",
    "    compartment_id=\"MY_OCID\",\n",
    ")\n",
    "\n",
    "resp = llm.stream_chat(messages)\n",
    "for r in resp:\n",
    "    print(r.delta, end=\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc2414f5",
   "metadata": {},
   "source": [
    "## Async"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f82bc4d7",
   "metadata": {},
   "source": [
    "Native async currently not supported. Async calls will revert to synchronous"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0d3e1b75",
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.llms.oci_genai import OCIGenAI\n",
    "from llama_index.core.llms import ChatMessage\n",
    "\n",
    "messages = [\n",
    "    ChatMessage(\n",
    "        role=\"system\", content=\"You are a pirate with a colorful personality\"\n",
    "    ),\n",
    "    ChatMessage(role=\"user\", content=\"Tell me a story\"),\n",
    "]\n",
    "\n",
    "llm = OCIGenAI(\n",
    "    model=\"MY_MODEL\",\n",
    "    service_endpoint=\"https://inference.generativeai.us-chicago-1.oci.oraclecloud.com\",\n",
    "    compartment_id=\"MY_OCID\",\n",
    ")\n",
    "\n",
    "resp = llm.achat(messages)\n",
    "print(resp)\n",
    "\n",
    "resp = llm.astream_chat(messages)\n",
    "for r in resp:\n",
    "    print(r.delta, end=\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "009d3f1c-ef35-4126-ae82-0b97adb746e3",
   "metadata": {},
   "source": [
    "## Configure Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e973e3d1-a3c9-43b9-bee1-af3e57946ac3",
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.llms.oci_genai import OCIGenAI\n",
    "\n",
    "llm = OCIGenAI(\n",
    "    model=\"cohere.command\",\n",
    "    service_endpoint=\"https://inference.generativeai.us-chicago-1.oci.oraclecloud.com\",\n",
    "    compartment_id=\"MY_OCID\",\n",
    ")\n",
    "\n",
    "resp = llm.complete(\"Paul Graham is \")\n",
    "print(resp)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1bdd4602-e37c-4230-af82-35af5292f9a0",
   "metadata": {},
   "source": [
    "## Authentication\n",
    "The authentication methods supported for LlamaIndex are equivalent to those used with other OCI services and follow the __[standard SDK authentication](https://docs.oracle.com/en-us/iaas/Content/API/Concepts/sdk_authentication_methods.htm)__ methods, specifically API Key, session token, instance principal, and resource principal.\n",
    "\n",
    "API key is the default authentication method. The following example demonstrates how to use a different authentication method (session token)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a9c80814-6d59-4782-a4bb-cbfcdba6a072",
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.llms.oci_genai import OCIGenAI\n",
    "\n",
    "llm = OCIGenAI(\n",
    "    model=\"MY_MODEL\",\n",
    "    service_endpoint=\"https://inference.generativeai.us-chicago-1.oci.oraclecloud.com\",\n",
    "    compartment_id=\"MY_OCID\",\n",
    "    auth_type=\"SECURITY_TOKEN\",\n",
    "    auth_profile=\"MY_PROFILE\",  # replace with your profile name\n",
    "    auth_file_location=\"MY_CONFIG_FILE_LOCATION\",  # replace with file location where profile name configs present\n",
    ")\n",
    "\n",
    "resp = llm.complete(\"Paul Graham is \")\n",
    "print(resp)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9845613a",
   "metadata": {},
   "source": [
    "## Dedicated AI Cluster\n",
    "To access models hosted in a dedicated AI cluster __[create an endpoint](https://docs.oracle.com/en-us/iaas/api/#/en/generative-ai-inference/20231130/)__ whose assigned OCID (currently prefixed by ‘ocid1.generativeaiendpoint.oc1.us-chicago-1’) is used as your model ID.\n",
    "\n",
    "When accessing models hosted in a dedicated AI cluster you will need to initialize the OCIGenAI interface with two extra required params (\"provider\" and \"context_size\")."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c5c153ed",
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.llms.oci_genai import OCIGenAI\n",
    "from llama_index.core.llms import ChatMessage\n",
    "\n",
    "llm = OCIGenAI(\n",
    "    model=\"ocid1.generativeaiendpoint.oc1.us-chicago-1....\",\n",
    "    service_endpoint=\"https://inference.generativeai.us-chicago-1.oci.oraclecloud.com\",\n",
    "    compartment_id=\"DEDICATED_COMPARTMENT_OCID\",\n",
    "    auth_profile=\"MY_PROFILE\",  # replace with your profile name,\n",
    "    auth_file_location=\"MY_CONFIG_FILE_LOCATION\",  # replace with file location where profile name configs present\n",
    "    provider=\"MODEL_PROVIDER\",  # e.g., \"cohere\" or \"meta\"\n",
    "    context_size=\"MODEL_CONTEXT_SIZE\",  # e.g., 128000\n",
    ")\n",
    "\n",
    "messages = [\n",
    "    ChatMessage(\n",
    "        role=\"system\", content=\"You are a pirate with a colorful personality\"\n",
    "    ),\n",
    "    ChatMessage(role=\"user\", content=\"Tell me a story\"),\n",
    "]\n",
    "\n",
    "resp = llm.chat(messages)\n",
    "print(resp)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "acd73b3d",
   "metadata": {},
   "source": [
    "## Basic tool calling in llamaindex \n",
    "\n",
    "Only Cohere supports tool calling for now"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea4b58c3",
   "metadata": {},
   "source": [
    "Use chat_with_tools for tool calling."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5546c661",
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.llms.oci_genai import OCIGenAI\n",
    "from llama_index.core.tools import FunctionTool\n",
    "\n",
    "llm = OCIGenAI(\n",
    "    model=\"MY_MODEL\",\n",
    "    service_endpoint=\"https://inference.generativeai.us-chicago-1.oci.oraclecloud.com\",\n",
    "    compartment_id=\"MY_OCID\",\n",
    ")\n",
    "\n",
    "\n",
    "def multiply(a: int, b: int) -> int:\n",
    "    \"\"\"Multiple two integers and returns the result integer\"\"\"\n",
    "    return a * b\n",
    "\n",
    "\n",
    "def add(a: int, b: int) -> int:\n",
    "    \"\"\"Addition function on two integers.\"\"\"\n",
    "    return a + b\n",
    "\n",
    "\n",
    "add_tool = FunctionTool.from_defaults(fn=add)\n",
    "multiply_tool = FunctionTool.from_defaults(fn=multiply)\n",
    "\n",
    "response = llm.chat_with_tools(\n",
    "    tools=[add_tool, multiply_tool],\n",
    "    user_msg=\"What is 3 * 12? Also, what is 11 + 49?\",\n",
    "    allow_parallel_tool_calls=True,\n",
    ")\n",
    "\n",
    "print(response)\n",
    "tool_calls = response.message.additional_kwargs.get(\"tool_calls\", [])\n",
    "print(tool_calls)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "63702c0e",
   "metadata": {},
   "source": [
    "Use chat for tool calling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "637eaeef",
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.llms.oci_genai import OCIGenAI\n",
    "from llama_index.core.llms import ChatMessage, MessageRole\n",
    "\n",
    "llm = OCIGenAI(\n",
    "    model=\"MY_MODEL\",\n",
    "    service_endpoint=\"https://inference.generativeai.us-chicago-1.oci.oraclecloud.com\",\n",
    "    compartment_id=\"MY_OCID\",\n",
    ")\n",
    "\n",
    "\n",
    "def multiply(a: int, b: int) -> int:\n",
    "    \"\"\"Multiple two integers and returns the result integer\"\"\"\n",
    "    return a * b\n",
    "\n",
    "\n",
    "def add(a: int, b: int) -> int:\n",
    "    \"\"\"Addition function on two integers.\"\"\"\n",
    "    return a + b\n",
    "\n",
    "\n",
    "add_tool = FunctionTool.from_defaults(fn=add)\n",
    "multiply_tool = FunctionTool.from_defaults(fn=multiply)\n",
    "\n",
    "user_msg = ChatMessage(\n",
    "    role=MessageRole.USER, content=\"What is 3 * 12? Also, what is 11 + 49?\"\n",
    ")\n",
    "\n",
    "response = llm.chat(\n",
    "    messages=[user_msg],\n",
    "    tools=[multiply_tool, add_tool],\n",
    ")\n",
    "\n",
    "print(response)\n",
    "tool_calls = response.message.additional_kwargs.get(\"tool_calls\", [])\n",
    "print(tool_calls)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
